atelet: serve shared snapshot restores from the node-local file cache - #1551
Draft
Dmitry Berkovich (dberkov) wants to merge 7 commits into
Draft
atelet: serve shared snapshot restores from the node-local file cache#1551Dmitry Berkovich (dberkov) wants to merge 7 commits into
Dmitry Berkovich (dberkov) wants to merge 7 commits into
Conversation
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 9, 2026 01:25
d0bdb37 to
240c6ce
Compare
Dmitry Berkovich (dberkov)
marked this pull request as draft
September 10, 2026 03:59
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 10, 2026 04:04
240c6ce to
667cd76
Compare
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 11, 2026 22:15
667cd76 to
e4dd730
Compare
RestoreRequest borrowed ExternalCheckpointConfiguration — a write destination — for the snapshot it reads, leaving read-side attributes of a restore source no place to live. Split the restore arm into its own ExternalRestoreConfiguration; the oneof's type change is wire-compatible (snapshot_uri keeps field number 1, and old peers decode the message as before). base_config supersedes golden_snapshot_uri, typing the DATA_ON_GOLDEN base as a restore source too. The old field stays until both components roll — callers set both and atelet prefers base_config — and is removed (and reserved) in a follow-up. Pure restructuring: no caller sets base_config yet, nothing reads it, and the wire bytes of every current request are unchanged.
Prefer RestoreRequest.base_config and fall back to the superseded golden_snapshot_uri, so ateapi and atelet can roll in either order. A transitional caller sets both; validation requires them to agree, and a base snapshot of either spelling is rejected outside SNAPSHOT_SCOPE_DATA_ON_GOLDEN scope.
Set RestoreRequest.base_config on both DATA_ON_GOLDEN resume paths (an external data snapshot and a local pause checkpoint combining with the golden), alongside the superseded golden_snapshot_uri it duplicates: an atelet from before base_config reads only the old field, so the dual-write lets the two components roll in either order. The old field and this duplication go away together in a follow-up once both sides have rolled.
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 11, 2026 22:33
e4dd730 to
03ae2e4
Compare
Whether a snapshot's files may be cached node-locally is the control plane's knowledge, not the node's: atelet should not infer it from URI layout or atespace conventions, which change as tags supersede golden snapshots. Add SnapshotSharing to the restore source message — deliberately the property (PRIVATE or SHARED) rather than the snapshot's kind, so new shared snapshot kinds need no atelet change — and set it in ateapi's resume workflow: SHARED when the restore reads the template's golden snapshot, PRIVATE for the actor's own snapshot. Nothing consumes the field yet.
Shared snapshot files (a template's golden snapshot today, tag snapshots later) are immutable once published, yet every restore that needs them re-downloads them into its own per-actor dir. Introduce the node-level cache that will hold them: a filecache.Store rooted under the ateom base path (cache hits will be served as hard links into per-actor restore dirs, which requires one mounted filesystem), opened at startup behind --snapshot-cache-dir with crash debris swept before atelet serves. No restore behavior changes yet: the store is wired into the herder but nothing reads through it. The cached download path and the eviction loop follow separately.
Snapshot files were downloaded into every actor's restore dir, even when a dozen actors resume from one shared snapshot on the same node. Route the cacheable legs of the restore download paths through the snapshot cache: the DATA_ON_GOLDEN base half (shared by construction), the local-pause combine leg, and an external restore whose source the control plane declared SHARED (a fresh-from-golden start). Private snapshots and requests from callers that predate the sharing field download fresh; atelet never infers cacheability from the URI. Cache hits are hard links to a read-only shared copy, so serving is gated per sandbox class: ateom-gvisor consumes restore-state strictly read-only and gets links; ateom-microvm rewrites config.json in place at restore and merges checkpoint deltas into memory-ranges' inode at suspend, either of which would corrupt a shared inode, so its class downloads fresh until a private-copy mode serves it.
The snapshot cache served only gVisor restores: its hits are read-only hard links, and ateom-microvm mutates staged files in place (config.json is rewritten at restore, checkpoint deltas are merged into memory-ranges' inode at suspend), which would corrupt a shared inode. Add cacheModeCopy: hits are hole-preserving private copies (see filecache.GetFileCopyTo), so the micro-VM class keeps its in-place mutations — no ateom-microvm changes — while concurrent restores of one snapshot still share a single download and later restores read the node cache instead of object storage. A link-mode fetch that fails with EXDEV (cache configured on a different mount than the restore dirs) now degrades to a copy too, keeping the download savings where it previously fell back to a fresh download per restore.
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 14, 2026 00:14
03ae2e4 to
251378a
Compare
Dmitry Berkovich (dberkov)
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements milestone M2 of the node-local artifact cache proposed in #690: atelet restores now serve shared snapshot files (a template's golden snapshot today, tag snapshots later) from a node-local cache backed by the
filecachestore from #1517, so concurrent restores of one snapshot share a single download and later restores read from disk instead of object storage.Stacked on #1602 (fork branches cannot be PR bases; #1517 is merged): please review only the top four commits — everything below
ateletpb, ateapi: declare how a restore's external snapshot is sharedis #1602, the Restore snapshot-source refactor this builds on.ateletpb, ateapi: declare how a restore's external snapshot is shared— cacheability is the control plane's knowledge, not the node's: atelet must not infer it from URI layout or atespace conventions, which change as tags supersede golden snapshots.SnapshotSharing(PRIVATE / SHARED) is deliberately the property rather than the snapshot's kind, so new shared snapshot kinds need no atelet change; ateapi's resume workflow sets SHARED when the restore reads the template's golden and PRIVATE for the actor's own snapshot. Nothing consumes the field yet.atelet: open a snapshot file cache at startup— lifecycle only:--snapshot-cache-dir(default<base-path>/snapshot-cache, empty disables; must share the actor dirs' mount because hits are hard links) and--snapshot-cache-min-age; the store opens at startup andSweepDebrisclears crash debris before atelet serves. No behavior change yet.atelet: serve shared snapshot downloads from the file cache— routes the cacheable legs through the cache: the DATA_ON_GOLDEN base half (shared by construction), the local-pause combine leg, and an external restore whose source is declared SHARED (a fresh-from-golden start). Private snapshots — and requests from callers predating the field — download fresh. Serving is per sandbox class: gVisor consumes restore-state read-only and gets zero-cost hard links.atelet: serve micro-VM snapshot downloads as private copies— micro-VM cannot share inodes (ateom-microvm rewritesconfig.jsonin place at restore and merges checkpoint deltas intomemory-ranges' inode at suspend), so its hits are hole-preserving private copies (GetFileCopyTo): one download per snapshot per node, mutations stay private, no ateom-microvm changes. A link-modeEXDEV(cache on a different mount) degrades to a copy too, keeping the download savings.The M2 exit criterion — N concurrent restores of one shared snapshot = one download — is pinned at the unit level by
TestDownloadExternalCheckpointSharesOneGoldenDownload(4 concurrent restores, 1 GCS get, shared inode) and its copy-mode counterpart; the sharing gate has its own matrix test, and a functional test drives the golden-data resume end to end asserting the control plane declares SHARED on the fresh resume and PRIVATE on the actor's own snapshot.Still to come for M2 (will follow once this lands): the pressure-driven GC loop for the snapshot cache (mirroring
imagegc's watermark formula) — until then the cache grows unbounded, which matches today's behavior forstatic-filesand is bounded in practice by golden churn.Tested:
go test -race ./cmd/atelet/...plus the controlapi functional suite; every commit builds and passes tests individually; the cross-mount EXDEV fallback has a dedicated test that exercises two real filesystems where available (/dev/shmon Linux CI) and skips elsewhere;golangci-lint, gofmt, and boilerplate checks clean.🤖 Generated with Claude Code